<html>
<head><meta charset="utf-8"><title>preallocation in serde · wg-secure-code · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/index.html">wg-secure-code</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html">preallocation in serde</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="174383744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174383744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Stuart Small <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174383744">(Aug 28 2019 at 17:24)</a>:</h4>
<p>I bring this up because of <a href="https://github.com/3Hren/msgpack-rust/issues/151" target="_blank" title="https://github.com/3Hren/msgpack-rust/issues/151">https://github.com/3Hren/msgpack-rust/issues/151</a> but I've seen this a couple times now.  It seems to be a reoccurring bug in serde crates that have a binary stream with the style of [length][datastream].   I fixed a similar bug in the bson crate but luckily with that one the spec has a small maximum size defined.. I looked at fixing the msgpack one with <a href="https://github.com/rust-lang/rust/issues/48043" target="_blank" title="https://github.com/rust-lang/rust/issues/48043">https://github.com/rust-lang/rust/issues/48043</a> but that might not be the best path since an attacker can still use the bug to gobble RAM but just won't be able to hit the same DOS panic bug.  Does anyone have a better approach to this class of bugs?  Should I look to the serde comunity?</p>



<a name="174384833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174384833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174384833">(Aug 28 2019 at 17:37)</a>:</h4>
<p>hahahaha</p>



<a name="174384844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174384844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174384844">(Aug 28 2019 at 17:38)</a>:</h4>
<p>deep rabbit hole</p>



<a name="174384889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174384889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174384889">(Aug 28 2019 at 17:38)</a>:</h4>
<p>some would argue this is a problematic family in formal language theory</p>



<a name="174384902"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174384902" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174384902">(Aug 28 2019 at 17:38)</a>:</h4>
<p>it wasn't even properly classified until a few years ago</p>



<a name="174384926"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174384926" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174384926">(Aug 28 2019 at 17:38)</a>:</h4>
<p><a href="http://spw17.langsec.org/papers/grosch-taming-length-fiels.pdf" target="_blank" title="http://spw17.langsec.org/papers/grosch-taming-length-fiels.pdf">http://spw17.langsec.org/papers/grosch-taming-length-fiels.pdf</a></p>



<a name="174384973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174384973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Stuart Small <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174384973">(Aug 28 2019 at 17:39)</a>:</h4>
<p>Thanks for the link on more reading</p>



<a name="174384975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174384975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174384975">(Aug 28 2019 at 17:39)</a>:</h4>
<p><a href="/user_uploads/4715/hSt7a25I1EXO92moXddkUH6a/Screen-Shot-2019-08-28-at-10.39.13-AM.png" target="_blank" title="Screen-Shot-2019-08-28-at-10.39.13-AM.png">Screen-Shot-2019-08-28-at-10.39.13-AM.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/hSt7a25I1EXO92moXddkUH6a/Screen-Shot-2019-08-28-at-10.39.13-AM.png" target="_blank" title="Screen-Shot-2019-08-28-at-10.39.13-AM.png"><img src="/user_uploads/4715/hSt7a25I1EXO92moXddkUH6a/Screen-Shot-2019-08-28-at-10.39.13-AM.png"></a></div>



<a name="174385003"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385003" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385003">(Aug 28 2019 at 17:39)</a>:</h4>
<p>it's a really good read if you're interested in this particular problem</p>



<a name="174385092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385092">(Aug 28 2019 at 17:40)</a>:</h4>
<p>lots of language theory, but they do boil it down to some practical advice about how to parse this family of languages</p>



<a name="174385204"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385204" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385204">(Aug 28 2019 at 17:42)</a>:</h4>
<p>in general I would say making efficient zero-copy parsers for these sorts of data formats is very much a classical source of vulnerabilities</p>



<a name="174385263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385263">(Aug 28 2019 at 17:42)</a>:</h4>
<p>and there are tons of sharp edges that are rarely considered</p>



<a name="174385335"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385335" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Stuart Small <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385335">(Aug 28 2019 at 17:43)</a>:</h4>
<p>Looking at this in the larger context of serde is making a lot of this sharp edges jump to the forefront too.</p>



<a name="174385427"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385427" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385427">(Aug 28 2019 at 17:44)</a>:</h4>
<p>yeah, I think in general <code>serde</code> is designed with context-free grammars in mind</p>



<a name="174385435"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385435" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385435">(Aug 28 2019 at 17:44)</a>:</h4>
<p>or if not that, "packed structs"</p>



<a name="174385485"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385485" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385485">(Aug 28 2019 at 17:45)</a>:</h4>
<p>I've written a serde parser/serializer for a format that's a combination thereof (which I didn't design), although that makes things a little easier: packed struct in front, with some messages having a variable length (length-prefixed) payload at the end</p>



<a name="174385543"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385543" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385543">(Aug 28 2019 at 17:46)</a>:</h4>
<p>that avoids nested length fields, which are where things generally start to go awry</p>



<a name="174385847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Stuart Small <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385847">(Aug 28 2019 at 17:49)</a>:</h4>
<p>Makes sense.  For most of these I'm looking the format is already defined and I just want to be able to handle safely.  I was thinking about peeking ahead to see if the length value in the protocol is longer than what is left in the stream.  Last I played with this was in the winter but IIRC serde didn't make that easy.  This is all just coming up again because of that thread getting bumped.</p>



<a name="174385946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174385946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Stuart Small <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174385946">(Aug 28 2019 at 17:50)</a>:</h4>
<p>I think I need to do some reading and catch back on what I was playing with last time I looked at this.  It makes me feel better that this is an active research problem.</p>



<a name="174415424"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174415424" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174415424">(Aug 29 2019 at 01:09)</a>:</h4>
<p>This isn't really just a Rust problem -- fuzzing ImageMagick/GraphicsMagick turned up tons of ooms of the shape "allocate however many X the length prefix said!"</p>



<a name="174428825"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174428825" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174428825">(Aug 29 2019 at 06:58)</a>:</h4>
<blockquote>
<p><a href="http://spw17.langsec.org/papers/grosch-taming-length-fiels.pdf" target="_blank" title="http://spw17.langsec.org/papers/grosch-taming-length-fiels.pdf">http://spw17.langsec.org/papers/grosch-taming-length-fiels.pdf</a></p>
</blockquote>
<p>thanks for sharing!</p>



<a name="174428893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174428893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174428893">(Aug 29 2019 at 06:58)</a>:</h4>
<p>funny that they claim "(1) Length-prefix languages arenot context-free" is a contribution of theirs, AFAIK that as well-known and I saw talks about it many years ago... but it might be just clumsy wording in the abstract</p>



<a name="174428918"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174428918" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174428918">(Aug 29 2019 at 06:59)</a>:</h4>
<p>oh wait, they claim to disprove (1)? Now I am confused.^^</p>



<a name="174429021"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429021" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429021">(Aug 29 2019 at 07:01)</a>:</h4>
<p><span class="user-mention" data-user-id="120791">@RalfJ</span> it's a little more nuanced than that. If you look at the image I pasted above (Fig 3), their novel contribution is that "Calc-Regular" languages exist outside the normal confines of the Chomsky Hierarchy</p>



<a name="174429094"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429094" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429094">(Aug 29 2019 at 07:02)</a>:</h4>
<p>that makes sense</p>



<a name="174429098"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429098" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429098">(Aug 29 2019 at 07:02)</a>:</h4>
<p>but then what do they mean by "disproving (1)"...</p>



<a name="174429113"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429113" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429113">(Aug 29 2019 at 07:02)</a>:</h4>
<p>oh, they mean the "parsing is inefficient" part</p>



<a name="174429119"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429119" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429119">(Aug 29 2019 at 07:02)</a>:</h4>
<p>got it :D</p>



<a name="174429135"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429135" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429135">(Aug 29 2019 at 07:03)</a>:</h4>
<p>interpreting a "Calc-Regular" language requires a sort of "recursive regular" automaton</p>



<a name="174429285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429285">(Aug 29 2019 at 07:06)</a>:</h4>
<p>which first parses and comprehends the length prefix and then uses that as its Kleene star when consuming the message, rinse repeat. but in addition to that, those sort of recursive sub-machines allow you to recursively descend into nested length-prefix data, providing properties closer to a context-free language, but without the need for a pushdown automaton, i.e. while it's popular to use a stack machine for comprehending nested length prefix data, the paper shows it isn't strictly necessary</p>



<a name="174429307"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429307" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429307">(Aug 29 2019 at 07:07)</a>:</h4>
<p>cool</p>



<a name="174429373"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174429373" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174429373">(Aug 29 2019 at 07:08)</a>:</h4>
<blockquote>
<p>Calc-regular languages are almost as easy to parse as regular languages, using finite state machines with additional accumulators.</p>
</blockquote>



<a name="174475955"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174475955" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Shnatsel <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174475955">(Aug 29 2019 at 17:50)</a>:</h4>
<p>AFAIK this is basically unsolved as of right now. Every single image or audio decoder has also bumped into this, see e.g. <a href="https://libpng.sourceforge.io/decompression_bombs.html" target="_blank" title="https://libpng.sourceforge.io/decompression_bombs.html">https://libpng.sourceforge.io/decompression_bombs.html</a><br>
So far the approach in C libraries is "put checks everywhere and hope you got them all and nothing ever overflows"<br>
I've found a number of such issues in Lewton, the Vorbis decoder in Rust, and the author came up with an interesting idea: make the memory allocator keep track of memory used by the crate and panic or abort or return error if someone tries to exceed the limit. This way deserialization of crafted input fails safely and in an isolated manner, without impacting the rest of the application. Here's their implementation of that approach, but sadly it's not complete enough to be actually used in lewton: <a href="https://github.com/est31/balloc" target="_blank" title="https://github.com/est31/balloc">https://github.com/est31/balloc</a><br>
More info on the issue and their approach: <a href="https://github.com/RustAudio/lewton/issues/34" target="_blank" title="https://github.com/RustAudio/lewton/issues/34">https://github.com/RustAudio/lewton/issues/34</a></p>



<a name="174476191"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476191" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476191">(Aug 29 2019 at 17:53)</a>:</h4>
<p>there's a simple solution. it's just not always applicable.</p>



<a name="174476259"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476259" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476259">(Aug 29 2019 at 17:54)</a>:</h4>
<p>make the maximum field length a knob in the parser</p>



<a name="174476435"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476435" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Shnatsel <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476435">(Aug 29 2019 at 17:55)</a>:</h4>
<p>Well, that's what they're trying to do, but actually <em>implementing</em> that is a game of whack-a-mole.</p>



<a name="174476444"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476444" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476444">(Aug 29 2019 at 17:56)</a>:</h4>
<p><a href="https://github.com/clasp-lang/veriform/blob/master/rust/src/parser.rs#L18" target="_blank" title="https://github.com/clasp-lang/veriform/blob/master/rust/src/parser.rs#L18">https://github.com/clasp-lang/veriform/blob/master/rust/src/parser.rs#L18</a></p>



<a name="174476464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476464">(Aug 29 2019 at 17:56)</a>:</h4>
<p>heh</p>



<a name="174476511"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476511" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476511">(Aug 29 2019 at 17:56)</a>:</h4>
<p>I'd like to rewrite that into a zero copy pull parser based on iterators</p>



<a name="174476536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476536">(Aug 29 2019 at 17:56)</a>:</h4>
<p>using the techniques in the Calc-Regular paper</p>



<a name="174476610"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476610" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Shnatsel <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476610">(Aug 29 2019 at 17:57)</a>:</h4>
<p>libpng has a knob, a bunch of Rust imaging crates got a knob after I bugged the maintainers about it, but it's still not very reliable because you need to get all the math right, never overflow, and not forget a single place with a single check. That's like avoiding buffer overflows in C all over again.</p>



<a name="174476671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174476671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Shnatsel <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174476671">(Aug 29 2019 at 17:58)</a>:</h4>
<p><a href="https://github.com/est31/balloc" target="_blank" title="https://github.com/est31/balloc">https://github.com/est31/balloc</a> is supposed to handle that for you transparently and reliably... or at least that's the idea. I don't think it's mature enough for production use.</p>



<a name="174477189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174477189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174477189">(Aug 29 2019 at 18:02)</a>:</h4>
<p><span class="user-mention" data-user-id="127617">@Shnatsel</span> my parser neatly sidesteps all that. unfortunately the solution means it isn't applicable to those sorts of problems (max overall message size cap which defaults to 1kB <span aria-label="stuck out tongue wink" class="emoji emoji-1f61c" role="img" title="stuck out tongue wink">:stuck_out_tongue_wink:</span>)</p>



<a name="174477215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174477215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174477215">(Aug 29 2019 at 18:02)</a>:</h4>
<p>I could probably turn it up just a wee bit, heh</p>



<a name="174477241"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174477241" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Shnatsel <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174477241">(Aug 29 2019 at 18:03)</a>:</h4>
<p>1kb compressed or decompressed?</p>



<a name="174477610"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174477610" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174477610">(Aug 29 2019 at 18:06)</a>:</h4>
<p>there is no compression <span aria-label="stuck out tongue wink" class="emoji emoji-1f61c" role="img" title="stuck out tongue wink">:stuck_out_tongue_wink:</span></p>



<a name="174477629"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/preallocation%20in%20serde/near/174477629" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/preallocation.20in.20serde.html#174477629">(Aug 29 2019 at 18:06)</a>:</h4>
<p>it's for credentials</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>